Surgical system including a non-white light general illuminator

ABSTRACT

An apparatus may configure an illuminator to illuminate a scene with non-white light. The illuminator includes a plurality of color component illumination sources. The non-white light is light other than white light and is generated by a combination of light output by the plurality of color component illumination sources. The apparatus may also control a camera to capture, in a plurality of color channels of the camera, a frame of the scene illuminated with the non-white light and adjust pixel values of a color channel of the camera in the frame of the scene to decrease noise of the color channel in the frame of the scene.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/158,976, filed Jan. 26, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/682,940, filed Nov. 13, 2019 and issued as U.S.Pat. No. 10,932,649, which application is a continuation of U.S. patentapplication Ser. No. 15/126,978, filed Sep. 16, 2016 and issued as U.S.Pat. No. 10,506,914, which is the U.S. national phase of InternationalApplication No. PCT/US2015/020893, filed Mar. 17, 2015, which claimspriority to U.S. Provisional Patent Application No. 61/954,512, filedMar. 17, 2014, all of which are incorporated herein by reference intheir entirety.

BACKGROUND Field of Invention

Aspects of this invention are related to endoscopic imaging and are moreparticularly related to non-white light used for general illumination ina teleoperated surgical system.

Related Art

The da Vinci® Surgical System, commercialized by Intuitive Surgical,Inc., Sunnyvale, California, is a minimally invasive teleoperatedsurgical system that offers patients many benefits, such as reducedtrauma to the body, faster recovery and shorter hospital stay. Onefeature of the da Vinci® Surgical System is a capability to providetwo-channel (i.e., left and right) video capture and display of visibleimages to provide stereoscopic viewing for the surgeon.

Such electronic stereoscopic imaging systems may output high definitionvideo images to the surgeon, and may allow features such as zoom toprovide a “magnified” view that allows the surgeon to identify specifictissue types and characteristics, as well as to work with increasedprecision. In a typical surgical field, however, the quality of theimage captured by a camera in the electronic stereoscopic imaging systemis limited by the signal-to-noise ratio of the camera.

As a camera collects light, the captured light is converted to electronsand stored in wells of an image sensor. There is one well per pixel.FIG. 1 is a schematic illustration of a well 101 for a red pixel R, awell 102 for a green pixel G, and a well 103 for a blue pixel B. As acamera collects more electrons into its wells, the signal grows whilethe noise stays relatively constant, and so the signal-to-noise ratioincreases, i.e. the signal captured in the well increases with respectto the noise.

A physical property of light capture by a camera is that the more lighta camera pixel captures, the better the camera can estimate the rate atwhich the light was captured. However, if a camera pixel collects toomuch light and overfills a well, the signal for that pixel is lost andno longer valid. Therefore, an exposure time of the camera is set to tryand collect light to fill all of its electron wells 101, 102, 103 ashigh as possible without overfilling any one well.

In a typical surgical site scene that is illuminated by white light forgeneral observations, red is the predominant color in the scene capturedby a camera. This because most of the reflected light is in the redspectrum relative the blue and green spectrums.

Typically, a color video camera used in a teleoperated surgical systemincludes a color filter array. The color filter array is a mosaic ofdifferent colored filters. Ideally, each different color filter passesonly a portion of the visible electromagnetic spectrum corresponding tothe spectrum of a particular color, e.g., a first set of filters in thecolor filter array passes primarily red light, a second set of filterspass primarily green light, and a third set of filters pass primarilyblue light.

The camera includes an image sensor that includes pixels that capturethe light that passes through the color filter array. Each pixel is awell that fills up with electrons as the light is captured. The set ofpixels in the camera that capture the light that passes through thefirst set of filters are included in a first color channel of thecamera. The set of pixels in the camera that capture the light thatpasses through the second set of filters are included in a second colorchannel of the camera. The set of pixels in the camera that capture thelight that passes through the third set of filters are included in athird color channel of the camera.

As is known to those knowledgeable in the field, in one example, whitelight illumination is made up of a combination of red spectrum light,green spectrum light and blue spectrum light that looks white to theeyes of a human with normal color perception. However, due to thepredominant reflection of the red spectrum light by the surgical site,red pixel well 101 (FIG. 1 ) typically fills up much faster than eithergreen pixel well 102, or blue pixel well 103. To prevent red pixel well101 from overflowing, the exposure of the camera is set to limit thelight collected so that red pixel well 101 does not overflow.

The consequence of stopping the collection of light when the wells ofthe color channel receiving the most light are about to overflow is thatthe wells of the other color channels may not be full as illustrated inFIG. 1 . In the example of FIG. 1 , green well 102 and blue well 103 areless than fifty-percent full when the collection of light is stopped.The signal-to-noise ratio of these less-full color channels issignificantly less than the signal-to-noise ratio of the color channelor channels that were about to overflow. Again, for the example of FIG.1 , the signal-to-noise ratio of the red channel is about six, while thesignal-to-noise ratio of each of the green and blue channels is aboutthree.

A camera has worse signal-to-noise ratio performance when not all ofwells 101, 102, 103 of the camera color channels are full. The signalsfrom less full wells 102 and 103 must have a gain applied to the signalsas part of a white balance stage in the surgical system's imageprocessing to create an image for display. White balancing is necessaryto ensure that when a camera captures an image of a white surface, thewhite surface appears white on the display monitor. White balancingconsists of amplifying less-full color channels (the blue and greencolor channels in FIG. 1 ), e.g., applying a digital gain, such thatthat all the color channels have equal values when the camera capturesan image of a white surface. The amplification of these less-full wellsignals increases the noise of these color signals relative to the othercolor signals, which further increases the noise in the final image.

SUMMARY

In one aspect, non-white light from an endoscope of a teleoperatedsurgical system is used to illuminate a surgical site. A camera capturesan image of the surgical site, and the image is displayed on a monitor.The non-white light illumination minimizes noise in the images of thesurgical site presented on the monitor relative to images captured usingwhite light illumination and displayed on the monitor.

While the color of the light used to illuminate the surgical site isnon-white light, e.g., light that has a purple tint, images displayed onthe monitor do not contain this tint. To the viewer, the lightilluminating the surgical site, as viewed in the monitor, looks white.Only if the endoscope is removed from the patient, and the light emittedfrom endoscope is viewed directly does one see the non-white light. Thenon-white light is used for general illumination and is different from,for example, a combination of only two narrow spectrum light sourcesused for highlighting specific anatomical structures.

In one aspect, an apparatus includes a camera and an illuminator. Thecamera is configured to separate light entering the camera into sets ofpixels. Each set of the sets of pixels being in a different colorchannel of the camera. In one aspect, the camera includes a color filterarray. The color filter array is configured to separate the lightentering the camera into the sets of pixels.

The illuminator is configured so that the illuminator outputs non-whitelight such that each camera color channel has an about equal response tothe non-white light reflected from a purely reflective surface. As usedhere, a purely reflective surface is a surface that has a response to anillumination spectrum that is spectrally uniform, equal attenuationacross the entire illumination spectrum. As used here, “about equal” or“substantially equal” means that the responses may not be exactly equaldue to differences in the reflective characteristics of the reflectivesurface (e.g., the reflective surface may not be precisely purelyreflective to the same extent everywhere on the surface) and due tonormal differences in the response of the electron wells of an imagesensor, but the responses are equal to within the combined tolerances ofthe image sensor and the surface.

In one aspect, the illuminator includes a plurality of color componentillumination sources. The plurality of color component illuminationsources is configured so that the illuminator outputs the non-whitelight.

In another aspect, the apparatus also includes a controller coupled tothe plurality of color component illumination sources. The controller isconfigured to weight the output of each of the plurality of colorcomponent illumination sources so that a combination of the outputs ofthe plurality of color component illumination sources is the non-whitelight.

In yet another aspect, an apparatus includes a camera, an illuminatorand a controller. The camera is configured to separate light enteringthe camera into color components. The color components are captured bythe camera as sets of pixels. Each set of the sets of pixels is in adifferent camera color channel. In one aspect, the camera includes acolor filter array. The color filter array is configured to separate thelight entering the camera into the sets of pixels.

The controller is coupled to the illuminator. The controller isconfigured to adjust a characteristic of light output by the illuminatorto increase a signal-to-noise ratio of pixels of one camera colorchannel for a color image captured by the camera.

In one aspect, the illuminator includes a plurality of color componentillumination sources. The controller is coupled to the plurality ofcolor component illumination sources. The controller is configured toadjust a characteristic of at least one of the plurality of colorcomponent illumination sources to increase the signal-to-noise ratio ofpixels of the one camera color channel.

In one aspect, the plurality of color component illumination sources isa plurality of light emitting diodes. In another aspect, the pluralityof color component illumination sources is a plurality of laser diodes.

In one aspect, the controller is configured to control an output of aplurality of color component illumination sources of the illuminator sothat the illuminator outputs non-white light such that each camera colorchannel has an about equal response to the non-white light reflectedfrom a purely reflective surface. In another aspect, the controller isconfigured to vary the illumination level of at least one of a pluralityof color component illumination sources of the illuminator so that theilluminator outputs non-white light. In another aspect, a fixed filteris used to vary the illumination level of at least one of a plurality ofcolor component illumination sources of the illuminator so that theilluminator outputs non-white light. In still another aspect, thevariation in illumination level to produce non-white light is controlledto adjust for unequal aging induced power loss of the life of theplurality of color component illumination sources.

In another aspect, the apparatus includes an image processing pipelineconfigured to create a high dynamic range image from a single colorimage captured by the camera. In this aspect, the controller isconfigured to vary the illumination level of at least one of a pluralityof color component illumination sources of the illuminator. The varyingof the illumination level can be implemented for example with a spinningwheel or a liquid crystal device.

The spinning wheel has a plurality of sections. Each of the plurality ofsections has a color of one of the plurality of color illuminationcomponents, and each of the plurality of sections has a different lightattenuation level.

In one aspect, the liquid crystal device is configured in an on/offpulse width modulation shutter mode with a variable ratio of on and offtime including one or more on / off cycles per camera image framecapture. In another aspect, the liquid crystal device is configured asan adjustable attenuator. In still another aspect, the liquid crystaldevice is configured as a wavelength tunable filter.

A method includes illuminating a scene with non-white light. Thenon-white light is configured so that each camera color channel of acamera has an about equal response to the non-white light reflected froma purely reflective surface. The method also includes capturing an imageof the scene with the camera, and outputting an image for display basedon the captured image without white color balancing of the capturedimage.

Another method includes capturing a color image. The captured colorimage includes sets of pixels. Each set of the sets of pixels being in adifferent color channel of the camera. This method also includesconstructing a high dynamic range image from the sets of pixels.

Still another method includes illuminating a site with non-white light.The non-white light being configured so that each camera color channelof a camera has an about equal response to non-white light reflectedfrom the site. This method also includes capturing an image of the sitewith the camera, and outputting an image for display based on thecaptured image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of the prior art fullness of electron well ofan image sensor for color components of an image captured of a sceneilluminated by white light.

FIG. 2 is a block diagram of a portion of a teleoperated surgical systemthat includes an illuminator that outputs non-white light.

FIG. 3 is an illustration of examples of responsivity functions of thecameras of FIG. 2 .

FIG. 4 is a graph of the power spectrum of each light emitting diode inone aspect of the illuminator of FIG. 2 .

FIG. 5 is a graph of a comparison of white light and non-white lightfrom the illuminator of FIG. 2 .

FIG. 6A is schematic diagram of a time line for generating dynamic colorchannel illumination control factors.

FIGS. 6B to 6D are illustrations of the fullness of electron well of animage sensor for color components of an image captured of a sceneilluminated by non-white light.

FIG. 7 is a diagram illustrating how to generate non-white light bydiffering relative on and off times over a time period of the outputs ofthe various color component illuminations sources from the illuminatorof FIG. 2 .

In the drawings, the first digit of a reference number indicates thefigure in which the element with that reference number first appeared.

DETAILED DESCRIPTION

As used herein, electronic stereoscopic imaging includes the use of twoimaging channels (i.e., channels for left and right images).

As used herein, a stereoscopic optical path includes two channels in anendoscope for transporting light from tissue (e.g., channels for leftand right images). The light transported in each channel represents adifferent view of the tissue. The light can form one or more images.Without loss of generality or applicability, the aspects described morecompletely below also could be used in the context of a field sequentialstereo acquisition system and/or a field sequential display system.

As used herein, an illumination path includes a path in an endoscopeproviding illumination to tissue.

As used herein, white light is visible white light that is made up ofthree (or more) visible color components, e.g., a red visible colorcomponent, a green visible color component, and a blue visible colorcomponent. White light may also refer to a more continuous spectrum inthe visible spectrum as one might see from a heated tungsten filament,for example.

As used herein, non-white light is visible light that is made up ofthree (or more) visible color components, e.g., a red visible colorcomponent, a green visible color component, and a blue visible colorcomponent in a combination that is different from the combination usedto make white-light. Non-white light may also refer to a more continuousspectrum in the visible electromagnetic spectrum, e.g., a broad spectrumof wavelengths in the visible electromagnetic spectrum that does notappear to a human viewer to be white light and that includes visiblespectrums of a plurality of colors. Non-white light does not include acombination of only two narrow spectrum light sources, such as acombination of two different narrow spectrum blue light sources or acombination of a narrow spectrum blue light source and a narrow spectrumgreen light source used to differentiate specific tissue.

As used herein, a color component has a spectrum of wavelengths withinthe visible electromagnetic spectrum.

As used herein, the visible electromagnetic spectrum ranges fromapproximately 400 nanometers (nm) to 700 nm in wavelength.

As used herein, a color image includes a combination of all of the colorcomponents of a color model in contrast to a monochromatic color imageor a color image that includes only a combination of a subset of thecolor components of the color model. For example, for a color model thatincludes red, green, and blue color components, a color image includes acombination of red, green, and blue color components. A red image, agreen image, a blue image, a blue and green image, etc. are not includedin the definition of a color image because such images do not include acombination of all of the color components of the color model.

In one aspect, light from an endoscope 201 of a portion of ateleoperated surgical system 200 is used to illuminate a surgical site203. The illumination is non-white light, e.g., the light looks purplishwhen viewed directly by a human. The use of non-white light illuminationminimizes noise in images of surgical site 203 presented on stereoscopicdisplay 251, sometimes referred to as display 251, in a surgeon'sconsole 250. Surgeon's console 250 is sometimes referred to as console250.

While the color of the light used to illuminate surgical site 203 isnon-white light, e.g., light that has a purple tint, images displayed onstereoscopic display 251 in surgeon's console 250 do not contain thistint. To the viewer, the light illuminating surgical site 203 as viewedthrough surgeon's console 250, looks white.

As explained more completely, below, cameras 220L, 220R and imageprocessing pipeline 240 in teleoperated surgical system 200 correct thecaptured images to remove the purple tint in surgical images displayedon stereoscopic display 251. It is only if the surgeon exits surgeon'sconsole 250, pulls endoscope 201 from the patient, and directly viewsthe light emitted from endoscope 201 does the surgeon see the non-whitelight.

Aspects of this invention facilitate illuminating surgical site 203 withnon-white illumination and facilitate acquiring color images of asurgical site 203 by cameras 220L, 220R (FIG. 2 ) in a teleoperatedsurgical system 200 with improved signal-to-noise ratios relative toimages captured using white light illumination of surgical site 203. Oneexample of a teleoperated surgical system 200 is the da Vinci® minimallyinvasive teleoperated surgical system commercialized by IntuitiveSurgical, Inc. of Sunnyvale, California. Teleoperated surgical system200 is illustrative only and is not intended to limit the application ofnon-white illumination to improve the signal-to-noise ratios of imagesto this specific teleoperated surgical system. In view of thisdisclosure, the non-white illumination can be used in any surgicalsystem that utilizes color cameras, or a color camera, to improve thesignal-to-noise ratio of the color images captured by those colorcameras.

In this example, a surgeon at surgeon's console 250 remotely manipulatesan endoscope 201 mounted on a robotic manipulator arm (not shown). Thereare other parts, cables, etc. associated with the da Vinci® SurgicalSystem, but these are not illustrated in FIG. 2 to avoid detracting fromthe disclosure. Further information regarding teleoperated minimallyinvasive surgical systems may be found for example in U.S. patentapplication Ser. No. 11/762,165 (filed Jun. 13, 2007; disclosingMinimally Invasive Surgical System) and U.S. Pat. No. 6,331,181 (filedDec. 18, 2001; disclosing Surgical Robotic Tools, Data Architecture, andUse), both of which are incorporated herein by reference.

An illumination system, e.g., illuminator 210, is coupled to endoscope201. In one aspect, illuminator 210 includes a light source 211 and anillumination controller 215. Illumination controller 215 is coupled tolight source 211 and to an optional variable non-white light apparatus218.

Illumination controller 215 includes a non-white light module 217 thatis connected between system process module 262 and light source 211.Non-white light module controls the output illumination from illuminator210 so that illuminator 210, in one aspect, outputs non-white light thatis used for general illumination of surgical site 203.

In one aspect, light source 211 includes a plurality of color componentillumination sources 212. In the aspect illustrated in FIG. 2 ,plurality of color component illumination sources includes P colorcomponent illuminations sources, where P is a non-zero positive integernumber. In one aspect, the number P is selected so that the combinationof the color component illumination sources provides the prior art broadspectrum white light. Also, to create non-white light, the outputoptical power of at least one of plurality of color componentillumination sources 212 is changed, either increased or decreased,relative to the state used to generate the prior art broad spectrumwhite light.

In one aspect, plurality of color component illumination sources 212includes a plurality of light emitting diodes (LEDs). The use of LEDs isillustrative only and is not intended to be limiting. Plurality of colorcomponent illumination sources 212 could also be implemented withmultiple laser sources instead of LEDs, for example.

In this aspect, illuminator 210 is used in conjunction with at least oneillumination path in stereoscopic endoscope 201 to illuminate surgicalsite 203. Non-white light from illuminator 210 is directed into aconnector 216. Connector 216 provides the non-white light to anillumination path in stereoscopic endoscope 201 that in turn directs thelight to surgical site 203. Each of connector 216 and the illuminationpath in stereoscopic endoscope 201 can be implemented, for example, witha fiber optic bundle, a single stiff or flexible rod, or an opticalfiber. Endoscope 201 also includes, in one aspect, two optical channels,i.e., a stereoscopic optical path, for passing light from surgical site203, e.g., reflected non-white light. However, use of a stereoscopicendoscope is illustrative only, and is not intended to be limiting. Inview of this disclosure, an endoscopic with a single optic channel forpassing light from surgical site 203 could be used.

The non-white light from surgical site 203 (FIG. 2 ) is passed by thestereoscopic optical channel in endoscope 201 to cameras 220L, 220R. Asexplained more completely below, in one aspect, left camera 220Lincludes a color filter array and a left image sensor 221L. Left imagesensor 221L captures the light received from the left channel ofstereoscopic endoscope 201 as a left image 222L. Similarly, in thisaspect right camera 220R includes a color filter array and a right imagesensor 221R. Right image sensor 221R captures the light received fromthe right channel of stereoscopic endoscope 201 as a right image 222R.Thus, cameras 220L, 220R are color cameras that use color filter arrays.However, this is illustrative only and is not intended to be limiting.

Herein, a camera is configured to separate light entering the camerainto N color components, the color components being captured by thecamera as N sets of pixels, each set of the sets of pixels being in adifferent camera color channel. Thus, each of cameras 220L, 220Rincludes a plurality of color channels. In one aspect, the plurality ofcolor channels is N color component channels, where N is a positivenon-zero integer number.

Camera 220L is coupled to a stereoscopic display 251 in surgeon'sconsole 250 by a left camera control unit 230L and image processingpipeline 240. Camera 220R is coupled to stereoscopic display 251 insurgeon's console 250 by a right camera control unit 230R and imageprocessing pipeline 240. Camera control units 230L, 230R receive signalsfrom a system process 262. System process 262 represents the variouscontrollers in system 200.

Display mode select switch 252 provides a signal to a user interface 261that in turn passes the selected display mode to system process 262.Various controllers within system process 262 configure non-white lightmodule 217 within illumination controller 215, configure left and rightcamera control units 230L and 230R to acquire the desired images, andconfigure any other elements in imaging processing pipeline 240 neededto process the acquired images so that the surgeon is presented therequested images in display 250. Imaging processing pipeline 240 isequivalent to known image processing pipelines, except for the detailsprovided herein.

Herein, the capture, processing, and display of images captured bycamera 220L is the same as the capture, processing, and display ofimages captured by camera 220R. Thus, in the following description, onlyimages associated with camera 220L are considered below. The descriptionis directly applicable to images associated with camera 220R and so thedescription is not repeated for camera 220R. The description is alsodirectly applicable to a system that utilizes only a single camera and asingle image processing pipeline with an endoscope that has a singleoptical channel.

As indicated previously, typically, three (or more) visible colorillumination components are combined to make white light, i.e., whitelight includes a combination of a first visible color component, asecond visible color component, and a third visible color component.Each of the three visible color components is a different visible colorcomponent, e.g., a red component, a green component, and a bluecomponent. More visible color illumination components may also be usedto create white light, such as a cyan component together with the red,green, and blue color components.

Also, as described above, light source 211, in one aspect, includes aplurality of color component illumination sources 212. To generatenon-white light illumination, in one aspect, non-white light module 217changes the output optical power of at least one of the plurality ofcolor component illumination sources 212 relative to what is requiredfor white light. The illumination output of non-illuminator 210 isnon-white light. Non-white light has a tint when compared to whitelight.

In FIG. 2 , cameras 220L, 220R and light source 212 are shown as beingexternal to endoscope 201. However, in one aspect, cameras 220L, 220Rand light source 212 are included in the distal tip of endoscope 201adjacent tissue 203.

There are different ways in which to configure illuminator 210 so thatilluminator outputs non-white light illumination. One way is based onwhite surface calibration, and a second way is based on a surgical siteimage calibration. In addition, generation of non-white light isdescribed that varies the duration of the illumination of at least oneof the plurality of color component illumination sources. Each of theseprocesses is considered in turn.

White Surface Calibration of Non-white Light

In this aspect, the non-white light mode works by illuminationcontroller 215 changing the illumination intensities from plurality ofcolor component illumination sources 212 in light source 211 so that allof a plurality of color channels of camera 220L receive the same amountof reflected light when viewing a white (perfectly reflective) surface.There are two beneficial effects of having camera 220L respond to lightevenly across color channels when imaging a white surface. First, allcolor channels can fully utilize their well capacity, i.e. no onechannel is restricted to a fraction of its well capacity due to anotherchannel filling its well first. This increases the signal-to-noise ofall pixels. Second, the camera does not have to apply a digital whitebalance gain to some color channels, which eliminates noiseamplification. Both of these effects enhance the final surgical imagesdisplayed on stereoscopic display 251.

Herein, noise properties of camera 220L are taken as being entirelydependent on how high the electron wells of image sensor 221L arefilled. As a camera 220L collects light, the captured light is convertedto electrons and stored in these electron wells. There is one electronwell per pixel. As image sensor 221L, e.g., camera 220L, collects moreelectrons in its electron wells, the camera's signal-to-noise ratioincreases, i.e. its signal relative to noise goes up.

As indicated previously, if camera 220L collects too much light andoverfills an electron well, the signal for that electron well is lostand no longer valid. Therefore, an exposure time of camera 220L is setto try and collect light to fill all of its electron wells as high aspossible without overfilling any electron well. The consequence ofstopping the collection of light when the color channel receiving themost light is about to overflow its electron wells is that the electronwells of the other color channels are not full. The electron wells ofthe other color channels may only be fifty percent full. Thesignal-to-noise ratio of the color channels having these less fullelectron wells is significantly less than the signal-to-noise ratio ofthe color channels with the electron wells that were about to overflow.

As described previously, for normal white light illumination, not onlydoes a camera have worse signal-to-noise ratio performance when not allof the camera's color channels have full or nearly full wells, but thesignals from the less full wells are amplified as part of the whitebalance stage in a prior art camera's image processing stage.

In contrast to normal white light illumination, one aspect of thenon-white light illumination mode works by changing the illuminationintensities of the plurality of color component illumination sources 212in light source 211 by illumination controller 215 so that all of theplurality of color channels in camera 220L receive the same amount oflight when viewing a white (perfectly reflective) surface. Since each ofthe plurality of color channels of camera 220L receives roughly the sameamount of light, the color channels of the camera can all be fullyutilized in the non-white illumination mode because no color channelreaches its well capacity first. Instead, the wells for the plurality ofcolor channels reach their well capacity at roughly the same time. Nocolor channel well will be at fifty percent of its well capacity whenimaging a white surface.

In addition, because the non-white illumination from illuminator 210 isdesigned to make all three camera channels respond to light evenly whenviewing a white surface, the white balance stage in imaging processingpipeline 240 used with white light illumination is not needed to amplifythe signals of any of the plurality of color channels. The signals forthe plurality of color channels are already equal. The overall effect issignificantly less noise in the resulting image as there is less noiseduring capture due to full-well utilization and there is noamplification of any color channels over other color channels.

To determine how to increase or decrease the intensity of the differentcolor component illumination sources in plurality of color componentillumination sources 212 so that the color channels of camera 220Lrespond equally when camera 220L, is viewing a white surface, thecharacteristics of camera 220L are considered and the control of lightsource 211 is considered. As noted previously, the considerations forcamera 220R are the same as those for camera 220L, and so thedescription is not repeated for camera 220R. Moreover, the followingdiscussion also applies to an endoscopic system that utilizes only asignal channel and a single camera.

Also, in the following discussion, a color space is considered thatutilizes red, green, and blue color components. Typically, a colorcamera includes a color filter array, such as a Bayer color filterarray. Irrespective of the configuration of the color camera, in thefollowing discussion, there is a first set of pixels captured by imagecapture sensor 221L that are associated with a red color channel of thecamera. There is a second set of pixels captured by image capture sensor221L that are associated with a green color channel of the camera, andthere is a third set of pixels captured by image capture sensor 221Lthat are associated with a blue color channel of the camera. The use ofthree color channels as representing the plurality of N color channelsof camera 220L is illustrative only and is not intended to be limiting.Also, the use of red, green, and blue color channels as the three colorchannels is illustrative only and is not intended to be limiting. Inview of this disclosure, one knowledgeable in the field can define boththe number of channels associated with camera 220L and the specificcolors associated with the number of channels based on a color space ofinterest and a color filter array of interest.

The optics of camera 220L, the color filter array used in camera 220L,and a quantum efficiency of camera 220L—collectively referred to as acamera's responsivity functions—determine how camera 220L responds todifferent wavelengths of incident light, i.e., responds to light fromsurgical site 203. FIG. 3 shows an example of responsivity functions forcamera 220L. There is a responsivity function for each of the pluralityof N color channels of camera 220L.

Thus, in this example, there are three responsivity functions, oneresponsivity function 301 for the blue color channel, one responsivityfunction 302 for the green color channel, and one responsivity function303 for the red color channel. In FIG. 3 , each of responsivityfunctions 301 to 303 is represented by a curve.

A higher value of a particular responsivity function indicates that acolor channel of camera 220L responds more to that particular wavelengthof light relative to other wavelengths of light that have a lower valuefor that particular responsivity function. For example, red responsivityfunction 303 shows that the red color channel camera 220L passes morelight in the 600 to 650 nanometer wavelength range than in the 450 to500 nanometer wavelength range.

A lower value of a particular responsivity function indicates that acolor channel of camera 220L responds less to that particular wavelengthof light relative to other wavelengths of light that have a higher valuefor that particular responsivity function. For example, blueresponsivity function 301 shows that the blue color channel of camera220L passes less light in the 600 to 650 nanometer wavelength range thanin the 440 to 470 nanometer wavelength range. A zero value on aresponsivity function indicates that the color channel of camera 220Lcannot see the wavelength of light.

In one aspect, the responsivity functions for the plurality of colorchannels in camera 220L are converted to matrix notation as three columnvectors that compose a matrix R. Matrix R is an M×N matrix.Specifically, in one aspect, a set of M uniformly-spaced wavelengths areselected, say from 400 nm to 700 nm spaced every 1 nm, and then thevalue of the responsivity function is read at each of these selectedwavelengths. In this example, 301 values are produced for eachresponsivity function. Thus, for this example, M equals 301 and N equals3 for the red, green and blue responsivity functions. As long as therange of 400 nm to 700 nm encompasses all important non-zero portions ofthe responsivity function, and the interval spacing is small enough (1nm), then the vector form is equivalent to a full curve. The samplingrange and interval usually changes based on the application.

As explained above, in one aspect, light source 211 includes a pluralityof P color component illumination sources 212. As an example, consideran implementation where P is four, so that light source 211 includesfour different color component illumination sources, e.g., fourindividual LEDs that can be adjusted to emit different intensities oflight. An example of an illuminator that includes four individual LEDsis shown in U.S. Patent Application Publication No. US 2012/0004508 Al(filed 2010 Aug. 13, disclosing “Surgical Illuminator With Dual SpectrumFluorescence”), which is incorporated herein by reference.

An example of an illumination spectrum of each of the four differentLEDs is shown in FIG. 4 . Spectrum 401 is blue color componentillumination. Spectrum 402 is a cyan color component illumination.Spectrum 403 is green color component illumination, and spectrum 404 isred color component illumination. In this aspect, each of the pluralityof color component illumination sources 212 is a different colorcomponent illumination source.

Spectra 401 to 404 can also be represented in matrix notation. Eachspectrum is a column of a matrix E. Matrix E is an M×P matrix.Specifically, in one aspect, a set of M uniformly-spaced wavelengths areselected, say from 400 nm to 700 nm spaced every 1 nm, and then thevalue of a LED illumination spectrum is read at each of these M selectedwavelengths.

Furthermore, because illumination controller 215 controls the outputintensity of each of the LEDs in light source 211 and because when theillumination outputs of the LEDS are optically combined together theoutputs mix linearly, the spectrum e^(out) of the light emitted from thedistal end of endoscope 201 can be represented as:

e ^(out)=diag(t)*E*w,

where

-   -   e^(out) is the emitted spectrum as a M×1 column vector,    -   t is the spectral transmission of the endoscope's illumination        channel represented as a M×1 column vector,    -   diag(x) denotes placing the vector x into the diagonal of a        matrix where all the other elements are zero, and    -   w is a P×1 single column weight vector and each element of        weight vector w is used by non-white light module 217 to        determine the intensity of a corresponding LED in the plurality        of P LEDs.

To achieve non-white light, a weight vector w is determined such thatthe response from each of the plurality of N camera color channels isequal subject to the constraint that the elements of weight vector wmust be positive as an LED cannot emit negative light. This can bewritten in matrix form:

[1]=R ^(T)*diag(t)*E*w subject to w>=0

where the goal is to solve for weight vector w, and [1] represents anN×1 column vector of ones, and R^(T) denotes an N×M matrix that is thematrix transpose of responsivity function matrix R.

Stated in another way, the previous expression determines P colorchannel illumination control factors CCIW (the elements in weight vectorw) such that when non-white light illumination module 217 applies adifferent one of P color channel illumination control factors CCIW toeach of the P color component illumination sources in light source 211,illuminator 210 outputs non-white light, and when the non-white lightemitted by endoscope 201 is incident on a purely reflective surface,i.e., a white surface, each of the electron wells of a camera 220L isfilled to a one hundred percent full level.

Thus, the problem is to find a weight vector w that includes onlypositive components. When the number P of controllable LEDs is equal tothe number N of camera color channels, a simple solution is just to takethe inverse as matrix R^(T)*diag(t)*E is a square matrix. In this case:

w=(R ^(T)*diag(t)*E)⁻¹*[1]

As long as weight vector w is all positive or zero (and R^(T)*diag(t)*Ehas an inverse), a solution exists. If weight vector w has negativecomponents, the LEDs cannot be controlled to equalize the camera colorchannels. But, in this situation, the negative components of weightvector w can be clipped to zero to get a solution that comes closest.

However, when the number P of controllable LEDs is not equal to thenumber N of camera color channels, a simple solution is not possiblebecause the inverse of matrix R^(T)*diag(t)*E does not exist. For theexample, with four controllable LEDs and camera 220L having only threecolor channels, a simple solution is not available.

In this particular case (or whenever the number P of controllable colorcomponent illumination sources is greater than the number N of colorchannels), there are many solutions for the components of weight vectorw, i.e. multiple different solutions to weight vector w will equalizethe filling of the wells of the camera color channels for non-whitelight incident on a purely reflective surface. Specifically, the set ofsolutions to w can be expressed as:

w=w ^(a) +V ^(n)*α by varying α subject to w>=0

where

-   -   w^(a) is the pseudo inverse solution,    -   V^(n) defines (P−N) null-space column vectors in a N×(P−N)        matrix, i.e. these are the directions you can vary the solution        w without changing the response of the camera, and    -   α defines one particular solution to w as a (P−N)×1 vector.        To limit the solution to one weight vector w, another constraint        must be imposed on the solution, i.e., we must specify the        null-space value by determining a single alpha value.

Possible constraints include but are not limited to maximizing emittedpower, minimizing emitted power, maximizing drive current, minimizingdrive current, or even changing the color of the light as viewed outsidethe surgeon's console to a more white appearance. Due to the constraintsof the LED drive electronics, a constraint was implemented thatmaximized the minimum LED intensity. Essentially, the desire is to havethe minimum value in weight vector w to be as high as possible becauseillumination controller 215 cannot control the light reliably at lowintensities.

There are minimax optimizations to solve for such constraints, but sincethere is only one extra degree of freedom in the solution (P−N=1) and itwas empirically noticed that the illumination from the green and cyanLEDs were always the lowest for camera 220L and they had opposite signsin the null space vector V^(n), the values in weight vector w weredetermined by sweeping a null space component alpha until the green andcyan channels were equal.

TABLE 1 is an example of Matlab computer code that is compiled on aMatlab compiler and then executed on a processor to determine weightvector w by sweeping a null space component alpha until the green andcyan illumination channels were equal. (MatLab is a U.S. registeredtrademark of The Mathworks, Inc., 3 Apple Hill Drive, Natick,Massachusetts 01760 U.S.A.)

TABLE 1 % Find the singular value decomposition of R^(T)*diag(t)*E [U,S, V] = svd(R^(T)*diag(t)*E) % Find the pinv solution. Vª = V(:,1:N) sª= diag(S(1:N,1:N)) w^(a) = V^(a)*diag(1·/sª)*U^(T)*1 % Find thenull-space component v^(n) = V(:,4) % Solve to make the green (index=3)and the cyan (index=2) equal. α = (wª (2)−w^(a) (3))·/(v^(n) (3)−v^(n)(2)) % Find the final w solution. w = w^(a) + v^(n+)αwhere

-   -   svd(X) is the singular value decomposition of X,    -   diag(x) is the diagonal matrix of x, and    -   the 2 & 3 indices correspond to the green and cyan LEDs        respectively.        Here, U, S, V are the outputs of the singular value        decomposition (svd). The singular value decomposition (svd) is a        common way to break up a matrix into three separate matrices:        X=USV^(T), where U and V are orthonormal matrices and S is a        diagonal matrix. This decomposition of X into U, S, V allows        finding the pseudo-inverse and the null space components.

FIG. 5 shows the difference between white light illumination 502 andnon-white illumination 501 using the above procedure. As evident infigure, the blue and red LEDs are outputting much more light than thegreen and cyan LEDs in the non-white light mode compared to white lightmode. In one aspect, the response of camera 220L to the white light whenbounced off of a white surface is R:61%, G:100% and B:67%, while thecamera response using the non-white light is R:100%, G:100% and B:100%as it was designed.

This technique of non-white light generation demonstrates that noisereduction in a displayed surgical scene can be achieved by adjusting thecolor of light from an endoscope. Specifically, the signal-to-noiseratio of each pixel can be increased in the displayed surgical scenewhen all the color channels of the camera or cameras capturing thatscene respond equally when viewing a white surface. As thesignal-to-noise ratio is increased, the perceived noise in the finalsurgical image decreases.

Scene-based Calibration of Non-white Light

In the prior example, non-white light module 217 was configured to drivelight source 211 so illuminator 210 illuminated surgical site 203 withnon-white light such that white light color balancing of the colorimages captured by camera 220L was unnecessary when the non-white lightis reflected by a white surface. This improved the signal-to-noise ratioof images displayed on display 251 relative to images captured usingnormal white light illumination and displayed on display 251.

However, non-white light can be generated in other ways to improve thesignal-to-noise ratio of images presented on display 251. For example,controller 215 is configured to vary the output illumination level,e.g., the output optical power, of at least one of the plurality ofcolor component illumination sources 212 so that illuminator 210 outputsnon-white light. The variation of the output illumination level of atleast one of the plurality of color component illumination sources 212is based on the color component characteristics of a color scenecaptured by camera 220L. The non-white light produced in this matteralso results in improved signal-to-noise ratio of scenes displayed ondisplay 251 relative to scenes captured using normal white lightillumination and displayed on display 251.

In one aspect, the color channel illumination control factors CCIW aregenerated in a different way. The information in a first captured sceneis used to generated color channel illumination control factors CCIWused by illumination controller 215.

In one aspect, a dynamic approach determines color channel illuminationcontrol factors CCIW that are used to generate non-white light. Forexample, a frame 610 (FIG. 6A) is captured and used to generate a firstset of color channel illumination control factors CCIW 620. The firstset of color channel illumination control factors CCIW 620 are used forthe next Z consecutive frames that are captured, where Z is a positivenon-zero integer number. While the Z consecutive frames are captured,processed, and displayed, one of the frames 621 in this sequence is usedto generate a second set of color channel illumination control factorsCCIW 621. The second set of color channel illumination control factorsCCIW 621 are used for the next Z consecutive frames that are captured,and the process continues.

In one aspect, a time weighted moving average is used to generate colorchannel illumination control factors CCIW 621. In this aspect, framechannel illumination control factors CCIW_frame are generated for eachframe, as described above, summed with a declining weighting over thenext fixed number of frames with other weighted frame channelillumination control factors CCIW_frame, and averaged so that framechannel illumination control factors CCIW_frame of the most recent framedominates but the (e.g. linearly) declining fractions of the framechannel illumination control factors CCIW_frame of previous frames aresummed to give a current applied time weighted moving average channelillumination control factors CCIW 621 updated at the frame rate, forexample.

In one aspect, number Z is determined empirically as a time average ofthe number of frames to maintain stability of system 200. In anotheraspect, the number Z is not a constant. Rather, the average brightnessin the color channels is monitored, and when the average brightness ofany one color channel changes by more than a predetermined percentage,e.g., five percent, a new set of color channel illumination controlfactors CCIW are generated. This approach adaptively compensates forchanges in the fullness of the wells in the camera's color channelduring the surgical procedure and adjusts the color channel illuminationcontrol factors CCIW so that the electron wells stay near the optimalfullness as the characteristics of the surgical site in the field ofview of the cameras change.

In another aspect, for the captured scene, a histogram of the brightnessof the pixels in each color channel of camera 220L is created. Ahistogram of the brightness of the pixels in each of the plurality of Ncolor channel of camera 220L is created. As is known to oneknowledgeable in the field, each pixel has a pixel value that is asingle number that represents the brightness of the pixel. The pixelvalue is also an indication of the fullness of the pixel well for thatpixel. Thus, for camera 220L, N brightness histograms are created—onefor each color channel.

In each of the N brightness histograms, the brightness values areplotted on an x-axis. A height of a bar for each of the brightnessvalues represents the number of pixels in the color channel having thatbrightness value. The brightness histogram can be based on the wholecaptured image, or on a region of interest in the captured image. Forexample, the region of interest could be defined as a region that iswithin the fovea of a person using teleoperated surgical system 200. Anindication of the fullness of the wells of the pixels of image sensor221L in that color channel can be derived from the brightness histogramfor each color channel is. The indication can be the mean value, or thevalue that corresponds to 90 percentile of all the values.

A matrix B is defined as the coupling between the illumination controlsand the camera sensor color channel responses. It transforms theillumination controls, a P element vector to the camera sensor colorchannel responses, an N element vector. Matrix B can be measured byturning on the illumination channels one at a time at a reference level.Thus,

$B = \begin{bmatrix}b_{11} & b_{12} & \ldots & b_{1P} \\b_{21} & b_{22} & \ldots & b_{2P} \\\ldots & \ldots & \ldots & \ldots \\b_{N1} & b_{12} & \ldots & b_{NP}\end{bmatrix}$

Color channel illumination control factors CCIW are defined as:

WFull=B*CCIW

B ⁻¹ *WFull=CCIW

whereWFull is a N×1 column vector with each component representing thedesired fullness of pixels wells in a color channel of camera 220L, and

-   -   CCIW is a P×1 column vector with each element representing a        color channel illumination control factor for one of the color        component illumination sources in light source 211.

If P equals N and an inverse of B exists, the determination of colorchannel illumination control factors CCIW is straight forward. If P islarger than N, a pseudo inverse of B is used.

A pseudo inverse of a matrix is known to those knowledgeable in thefield. A pseudo inverse suitable to use here is referred to as theMoore—Penrose pseudo inverse. A common use of the Moore-Penrose pseudoinverse is to compute a ‘best fit’ least squares solution to the systemof linear equations, which lacks a unique solution. Another use of theMoore-Penrose pseudo inverse is to find the minimum (Euclidean) normsolution to the system of linear equations. In one aspect, the best fitleast squares solution is used.

The color channel illumination control factors CCIW generated in thisway can be used in either a static approach or a dynamic approach, asdescribed above, in teleoperated surgical system 200. Irrespective ofthe technique used to generate the color channel illumination controlfactors CCIW, the use of non-white light illumination improves thequality of images displayed on displayed 251 by reducing thecontribution of noise to the displayed images relative to displayedimages that were created from images captured using white lightillumination.

As an example, assume that camera 220L has red, green, blue colorchannels, e.g., N is three, and desired well fullness WFull isseventy-five percent for each color channel. With white lightillumination, red channel R 603B is 90% full (FIG. 6B), green channel G602B is 50%, and blue channel B 601B is 37.5%. For non-whiteillumination, red channel R 603C is about 75% full (FIG. 6C), greenchannel G 602C is 75%, and blue channel B 601C is about 75% full. Thenoise floor for both white and non-white illuminations is taken as 15%.Thus, the signal to noise ratio of the blue and green color componentsis improved for non-white light.

However, because the illumination from each of the plurality of colorcomponent illumination sources 212 in light source 211 is absorbed andreflected differently by surgical site 203, the electron wells arelikely not exactly seventy-five percent full. Thus, in this aspect, whenthe responses of the camera channels to non-white light are said to beabout equal, it means that the responses of the portions of the camera'simage sensor in the different color channels are equal when thedifferences in absorption and reflection of the non-white light by thesurgical site are taken into consideration. Nevertheless, thesignal-to-noise ratio has been improved at least for blue and greenpixels.

In a color image of surgical site 203, the green and blue pixels providemost of the fine detail, because the green and blue color illuminationcomponents are scattered less and penetrate less than the red colorillumination component. Consequently, improving the signal-to-noiseratio of the blue and green pixels improves the quality of an imageprovide on display 251.

If pixels having the wells filled as illustrated in FIG. 6C wereprocessed directly for display on display 251, the displayed image wouldnot have the proper color balance due to the fullness of the electronwells in the color channels associated with the non-white lightillumination. Thus, when the pixels in each color channel are retrievedfrom the image sensor, the values of pixels in a color channel arecorrected to compensate for color channel illumination control factorsCCIW. Assuming illumination control L0 produces a white image when ascene of neutral color objects (e.g., a white balance target) isobserved,

I0=B*L0

where

-   -   I0=[r0,g0,b0]^(T) is an N×1 pixel brightness matrix where the        RGB components have equal values, and    -   L0=[L01, . . . , L0P]^(T) is a P×1 optical power output matrix.

At a later time an extra gain A=[ccw1, . . . , ccwP]^(T)=CCW^(T) isapplied to the illumination control on top of L0 resulting in L, whereL=[L1, . . . , LP]^(T)=[ccw1*L01, . . . , ccwP*L0P]^(T). The colorresponse from the camera with illumination L is I=[r,g,b]^(T).

K=B*A=[kr,kg,kb]

Adjusted pixel color I′=[r/kr, g/kg, b/kb]^(T) is presented to thedisplay to achieve the correct color. This results in a typical pixelvalue as illustrated in FIG. 6D. While the noise for the red colorchannel has increased a little, the noise for the green and blue colorchannels has deceased. Thus, when the signals in the green and bluecolor channels are amplified in the white color balance process in imageprocessing pipeline 240, the signal-to-noise ratio is better than whenusing white light illumination.

High Dynamic Range Images with Non-white Light

A standard way to create a high dynamic range image using a video camerais to have the video camera take consecutive images at differentexposure levels. The differently exposed images are merged into a singlehigh definition image. This technique requires that the video camerahave the ability to switch exposure settings frame by frame. Thisresults in an effectively reduced frame rate compared to the videocamera generating a display image from each captured frame. Also, ifthere is motion in the scene, when the images captured over time aremerged into the single high definition image, motion artifacts arelikely to be observed.

Another approach that is used in photography to create a high dynamicrange image is to capture an image using a graduated neutral densityfilter. The neutral density filter is graduated so that bright regionsof the scene are attenuated by the filter more than the dimmer regionsof the scene. While this works well for scenes having known regions ofdifferent brightness, e.g., scenes including a sunset or a sunrise, theuse of a graduated neutral density filter in a teleoperated surgicalsystem is not practical because the same portion of the scene of thesurgical site is not always the brightest region of the scene.

In applications that do not require a color image, camera 220L andnon-white light illumination of surgical site 203 can be used togenerate a monochromatic high dynamic range image. For example, unlikediagnosis, a color image may not be important during navigation of asurgical instrument or instruments, e.g., during guiding lungnavigation.

For applications in which a monochromatic image is acceptable, it ispossible to create a high dynamic range image using non-white lightillumination and a camera that is not made to be used with differentexposure settings or with a neutral density filter. Non-white lightmodule 217 is configured to drive plurality of color componentillumination sources 212 so that plurality of color componentillumination sources 212 have intensities other than the normalintensities used to create white light illumination.

Hence, camera 220L captures N images, one in each of N color channels ofcamera with effectively different exposures due to the different weightsused on the color component illumination sources 212. These N images areused to generate a high dynamic range image in a manner equivalent tothat used for a camera with a neutral density filter. Thus, a highdynamic range image is obtained without requiring any special filter ora camera with variable exposure settings. The use of non-white lightillumination allows the creation of a high dynamic range image using aconventional camera with a color filter array that is used inteleoperated surgical system.

For purposes of an example, assume that the number P of color componentillumination sources in plurality of color component illuminationsources 212 is three, and that the three color component illuminationsources are a red light source, a green light source, and a blue lightsource. Also, assume that for normal white light illumination, non-whitelight module 217 weights each of the three light sources equally, e.g.the red, green, and blue weights (color channel illumination controlfactors CCIW) are 1:1:1. In one aspect, the weights (color channelillumination control factors CCIW) are changed so that illuminator 210provides non-white light, e.g., for red, green and blue light sourcesthe weights are 0.5:1.0:2.0.

In general the weights of the color components are determined by takingthe ratio of the dynamic range of pixels in the color channels of camera220L (pixel dynamic range) to the dynamic range of the reflected lightfrom the surgical scene (scene dynamic range). The ratio of the pixeldynamic range to the scene dynamic range is 1:DR, where DR is a positivenon-zero integer. For this example, the largest weight for a firstillumination color component is DR times a weight for the Nthillumination color component. The weights for second through (N−1)illumination color components are uniformly spaced between the smallestweight and the largest by powers of two. For example if the pixeldynamic range is 1:256 and the scene dynamic range is 1:1024, the ratioof the two is 1:4 (2²). In the above example, the smallest weight was 1and so the largest weight was 4. The power of two between 1 and 4 is2¹=2 and so the third weight is 2.

As another example consider a ratio of 1:16 and three color componentillumination sources. The smallest weight is 1 and the largest weight is16. The other weight is 4. If the ratio is 1:16 and there are four colorcomponent sources, the weights would be 1:2^((4/3)):2^((8/3)):16.

When a scene is captured in camera 220L from reflected non-white light.Each of the N color channels captures a gray-scale scene with differentreflected light intensity due to the differences in the optical outputpower of the P color component illumination sources in light source 212.The N gray-scale scenes are processed to generate a high dynamic rangegray-scale image. One technique for doing this processing is describedin Nayar and Mitsunga, “High Dynamic Range Imaging: Spatially VaryingPixel Exposures,” IEEE Conference on Computer Vision and PatternRecognition, Vol. 1, pp. 472-479 (2000), which is incorporated herein byreference.

Rather than use images obtained with different camera exposure settingsor a neutral density filter, non-white light illumination comprising acombination of different intensity color illumination components is usedto obtain different exposure images with a fixed exposure camera. SinceN different exposure images are captured at the same time, there are nomotion artifacts.

The previous example used different intensity color components ofnon-white light to capture differently exposed images in a single frame.The same effect can be obtained by varying the length of time each colorcomponent illumination source is output within the exposure time of theframe. In this aspect, the weights applied by non-white light module 217are the same as those used for white light, but a switching element orelements are added to non-white light module 217.

Consider the same example, where the ratio of output optical power ofthe red, green, and blue color components is 0.5:1:2 and the exposuretime is a time t. For this example, the blue light source is output fora time (2/3.5)*t. The green light source is output for a time (1/3.5)*t,and the red light source is output for a time (0.5/3.5)*t. Thus, each ofthe color component light source is pulse width modulated to be on aspecified percentage of the exposure time.

FIG. 7 is a diagram of one example of varying the output from the red,blue, and green color component illumination sources in illuminator 210to control the output light from illuminator 210. The particular orderof varying each color component illumination source on and off is notimportant so long as for fifty-eight percent of the exposure time theblue light source is on, for twenty-eight percent of the exposure timethe green light source is on, and for fourteen percent of the exposuretime the red light source is on. Of course, instead of switching thelight sources on and off, the light source could be maintained in the onstate, and the output of the light source could be directed away from orblocked from reaching the output of illuminator 210 during the exposuretime as shown as off in FIG. 7 .

The particular number of light component sources and the weighting ofthe light component sources used in the above examples are illustrativeonly and are not intended to be limiting to the specific number of lightcomponent sources and weightings used in the example.

In another example, the output of the color component illuminationsources are varied as shown in FIG. 7 , but a different frame iscaptured for each color component illumination source. If a fixed frameshutter is used, the switching of the illumination is synchronized withimage acquisition. If a rolling shutter is used, the switching of theillumination must be at a frequency such that the switching illuminationdoes not introduce flicker into the captured image. As used here, arolling shutter means that instead of capturing the entire frame atonce, information is read from each row of the frame one after theother, e.g., from top to bottom.

Thus, in this example, the exposure time of the camera is fixed, but theillumination is varied to obtain differently exposed images. In thiscase, the high dynamic range image is formed in a manner that isequivalent to the known techniques for obtaining a high dynamic rangeimage for images taken at different exposures.

The varying of the output capability of illuminator 210 can beimplemented in a number of ways. The output of the plurality ofcomponent light sources can be directly controlled as described above.Element 218 can be a spinning wheel having a plurality of sections canbe placed in the path of the light output from illuminator 210. Each ofthe plurality of sections has a color of one of the plurality of colorcomponents, and each of the plurality of sections has a different lightattenuation level. In another aspect, element 218 is an acousto-opticlight modulator coupled to the controller.

Alternatively, element 218 can be a liquid crystal device. In oneaspect, the liquid crystal device is configured in an on/off pulse widthmodulation shutter mode with a variable ratio of on and off timeincluding one or more on/off cycles per camera image frame capture. Inanother aspect, the liquid crystal device is configured as an adjustableattenuator, e.g., an entrance polarizer, a compensated liquid crystalvariable retarder, and an exit polarizer, where the entrance and exitpolarizers are crossed linear polarizers. In still another aspect, theliquid crystal device is configured as a wavelength tunable filter. Theuse and operation of liquid crystal wavelength tunable filters are knownto those knowledgeable in the field.

Thus, in some aspects, a fixed filter is used to vary the illuminationlevel of at least one of the plurality of color component illuminationsources so that the illuminator outputs non-white light.

Also, in some aspects, the variation in illumination level to producenon-white light is controlled to adjust for unequal aging induced powerloss over the life of plurality of color component illumination sources212. Thus, as the optical output power a light emitting diode or laserdiode diminishes over time due to aging induced power losses, the colorchannel illumination control factors CCIW can be adjusted, eitherstatically or dynamically, to compensate for unequal aging induced powerlosses of each of plurality of color component illumination sources 212.

The various modules described herein can be implemented by softwareexecuting on a processor, hardware, firmware, or any combination of thethree. When the modules are implemented as software executing on aprocessor, the software is stored in a memory as computer readableinstructions and the computer readable instructions are executed on theprocessor. All or part of the memory can be in a different physicallocation than a processor so long as the processor can be coupled to thememory. Memory refers to a volatile memory, a non-volatile memory, orany combination of the two.

Also, the functions of the various modules, as described herein, may beperformed by one unit, or divided up among different components ordifferent modules, each of which may be implemented in turn by anycombination of hardware, software that is executed on a processor, andfirmware. When divided up among different components or modules, thecomponents or modules may be centralized in one location or distributedacross system 200 for distributed processing purposes. The execution ofthe various modules results in methods that perform the processesdescribed above for the various modules and controller 260.

Thus, a processor is coupled to a memory containing instructionsexecuted by the processor. This could be accomplished within a computersystem, or alternatively via a connection to another computer via modemsand analog lines, or digital interfaces and a digital carrier line.

Herein, a computer program product comprises a computer readable mediumconfigured to store computer readable code needed for any part of or allof the processes described herein, or in which computer readable codefor any part of or all of those processes is stored. Some examples ofcomputer program products are CD-ROM discs, DVD discs, flash memory, ROMcards, floppy discs, magnetic tapes, computer hard drives, servers on anetwork and signals transmitted over a network representing computerreadable program code. A non-transitory tangible computer programproduct comprises a tangible computer readable medium configured tostore computer readable instructions for any part of or all of theprocesses or in which computer readable instructions for any part of orall of the processes is stored. Non-transitory tangible computer programproducts are CD-ROM discs, DVD discs, flash memory, ROM cards, floppydiscs, magnetic tapes, computer hard drives and other physical storagemediums.

In view of this disclosure, instructions used in any part of or all ofthe processes described herein can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

The above description and the accompanying drawings that illustrateaspects and embodiments of the present inventions should not be taken aslimiting—the claims define the protected inventions. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, andtechniques have not been shown or described in detail to avoid obscuringthe invention.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms—such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike—may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of thedevice in use or operation in addition to the position and orientationshown in the figures.

For example, if the device in the figures is turned over, elementsdescribed as “below” or “beneath” other elements or features would thenbe “above” or “over” the other elements or features. Thus, the exemplaryterm “below” can encompass both positions and orientations of above andbelow. The device may be otherwise oriented (rotated 90 degrees or atother orientations) and the spatially relative descriptors used hereininterpreted accordingly.

Likewise, descriptions of movement along and around various axes includevarious special device positions and orientations. The singular forms“a”, “an”, and “the” are intended to include the plural forms as well,unless the context indicates otherwise. The terms “comprises”,“comprising”, “includes”, and the like specify the presence of statedfeatures, steps, operations, elements, and/or components but do notpreclude the presence or addition of one or more other features, steps,operations, elements, components, and/or groups.

Components described as coupled may be electrically or mechanicallydirectly coupled, or they may be indirectly coupled via one or moreintermediate components. In view of this disclosure, instructions usedin any one of, or any combination of operations described with respectto the augmented display system can be implemented in a wide variety ofcomputer system configurations using an operating system and computerprogramming language of interest to the user.

All examples and illustrative references are non-limiting and should notbe used to limit the claims to specific implementations and embodimentsdescribed herein and their equivalents. The headings are solely forformatting and should not be used to limit the subject matter in anyway, because text under one heading may cross reference or apply to textunder one or more headings. Finally, in view of this disclosure,particular features described in relation to one aspect or embodimentmay be applied to other disclosed aspects or embodiments of theinvention, even though not specifically shown in the drawings ordescribed in the text.

1. A system comprising: one or more processors; and memory storingexecutable instructions that, when executed by the one or moreprocessors, cause an apparatus to perform a process comprising:configuring an illuminator to illuminate a scene with non-white light,the illuminator comprising a plurality of color component illuminationsources and the non-white light comprising light other than white lightand generated by a combination of light output by the plurality of colorcomponent illumination sources; controlling a camera to capture, in aplurality of color channels of the camera, a frame of the sceneilluminated with the non-white light; and adjusting pixel values of acolor channel of the camera in the frame of the scene to decrease noiseof the color channel in the frame of the scene.
 2. The system of claim1, wherein the configuring the illuminator to illuminate the scene withthe non-white light is based on a signal level of the color channel ofthe camera in a previously captured frame of the scene.
 3. The system ofclaim 1, wherein: the configuring the illuminator to illuminate thescene with the non-white light comprises weighting an outputillumination level of one or more of the color component illuminationsources so that a combination of the light output by the plurality ofcolor component illumination sources is the non-white light; and theadjusting the color channel of the camera in the frame of the scene isbased on the weighting the output illumination level of the colorcomponent illumination source corresponding to the color channel of thecamera.
 4. The system of claim 1, wherein: a video stream captured bythe camera comprises a plurality of frames of the scene, the pluralityof frames including a first set of frames and a second set of framescaptured subsequently to the first set of frames; the second set offrames includes the frame of the scene and the first set of framesincludes a previously captured frame of the scene; and the configuringthe illuminator to illuminate the scene with the non-white lightcomprises: determining, based on color component characteristics oflight captured by the camera in one or more frames of the first set offrames, a first set of color channel illumination control factors, thefirst set of color channel illumination control factors comprising acolor channel illumination control factor for each color componentillumination source of the plurality of color component illuminationsources; and weighting, for each frame included in the second set offrames, output illumination levels of the plurality of color componentillumination sources based on the first set of color channelillumination control factors.
 5. The system of claim 4, wherein thefirst set of color channel illumination control factors is determinedbased on signal levels of the plurality of color channels for thepreviously captured frame of the scene.
 6. The system of claim 4,wherein the first set of color channel illumination control factors isdetermined based on a time-weighted moving average of color channelillumination control factors for each frame in the first set of frames.7. The system of claim 4, wherein the process further comprises:monitoring an average brightness of a color channel of the camera; anddetermining the first set of color channel illumination control factorsin response to a detection of a predetermined change of the averagebrightness of the color channel.
 8. The system of claim 4, wherein theadjusting the color channel of the camera in the frame of the scene todecrease the noise of the color channel is based on the first set ofcolor channel illumination control factors.
 9. The system of claim 1,wherein the color channel of the camera comprises a blue color channelor a green color channel.
 10. A method comprising: configuring, by acomputing device, an illuminator to illuminate a scene with non-whitelight, the illuminator comprising a plurality of color componentillumination sources and the non-white light comprising light other thanwhite light and generated by a combination of light output by theplurality of color component illumination sources; controlling, by thecomputing device, a camera to capture, in a plurality of color channelsof the camera, a frame of the scene illuminated with the non-whitelight; and adjusting, by the computing device, pixel values of a colorchannel of the camera in the frame of the scene to decrease noise of thecolor channel in the frame of the scene.
 11. The method of claim 10,wherein the configuring the illuminator to illuminate the scene with thenon-white light is based on a signal level of the color channel of thecamera in a previously captured frame of the scene.
 12. The method ofclaim 10, wherein: the configuring the illuminator to illuminate thescene with the non-white light comprises weighting an outputillumination level of one or more of the color component illuminationsources so that the combination of the light output by the plurality ofcolor component illumination sources is the non-white light; and theadjusting the color channel of the camera in the frame of the scene isbased on the weighting the output illumination level of the colorcomponent illumination source corresponding to the color channel of thecamera.
 13. The method of claim 10, wherein: a video stream captured bythe camera comprises a plurality of frames of the scene, the pluralityof frames including a first set of frames and a second set of framescaptured subsequently to the first set of frames; the second set offrames includes the frame of the scene and the first set of framesincludes a previously captured frame of the scene; and the configuringthe illuminator to illuminate the scene with the non-white lightcomprises: determining, based on signal levels of the plurality of colorchannels for the first set of frames in the video stream, a first set ofcolor channel illumination control factors, the first set of colorchannel illumination control factors comprising a color channelillumination control factor for each color component illumination sourceof the plurality of color component illumination sources; and weighting,for each frame included in the second set of frames, output illuminationlevels of the plurality of color component illumination sources based onthe first set of color channel illumination control factors.
 14. Themethod of claim 13, wherein the first set of color channel illuminationcontrol factors is determined based on signal levels of the plurality ofcolor channels for the previously captured frame of the scene.
 15. Themethod of claim 13, wherein the first set of color channel illuminationcontrol factors is determined based on a time-weighted moving average ofcolor channel illumination control factors for each frame in the firstset of frames.
 16. The method of claim 13, further comprising:monitoring, by the computing device, an average brightness of a colorchannel of the camera; and determining, by the computing device, thefirst set of color channel illumination control factors in response to adetection of a predetermined change of the average brightness of thecolor channel.
 17. The method of claim 13, wherein the adjusting thepixel values of the color channel of the camera in the frame of thescene is based on the first set of color channel illumination controlfactors.
 18. The method of claim 10, wherein the color channel of thecamera comprises a blue color channel or a green color channel.
 19. Anon-transitory computer-readable medium storing instructions that, whenexecuted, cause a processor of a computing device to perform a processcomprising: configuring an illuminator to illuminate a scene withnon-white light, the illuminator comprising a plurality of colorcomponent illumination sources and the non-white light comprising lightother than white light and generated by a combination of light output bythe plurality of color component illumination sources; controlling acamera to capture, in a plurality of color channels of the camera, aframe of the scene illuminated with the non-white light; and adjustingpixel values of a color channel of the camera in the frame of the sceneto decrease noise of the color channel in the frame of the scene. 20.The computer-readable medium of claim 19, wherein: the configuring theilluminator to illuminate the scene with the non-white light comprisesweighting an output illumination level of one or more of the colorcomponent illumination sources so that a combination of the light outputby the plurality of color component illumination sources is thenon-white light; and the adjusting the color channel of the camera inthe frame of the scene is based on the weighting the output illuminationlevel of the color component illumination source corresponding to thecolor channel of the camera.